‘wrangle’ defined

/ˈraŋɡl/

noun

a dispute or argument, typically one that is long and complicated. “an insurance wrangle is holding up compensation payments”

verb

1. have a long, complicated dispute or argument. “the bureaucrats continue wrangling over the fine print”

2. NORTH AMERICAN round up, herd, or take charge of (livestock). “the horses were wrangled early”

Wrangler

• Jeep Wrangler
• Wrangler Jeans
• Cowboys

Data Wrangling

Why tidy data?

• helps future you
  • and collaborators
• facilitates sharing your data and code (Laurinavichyute et al., 2022)
• in short: facilitates reproducibility!

What does tidy data look like?

Three rules (Wickham et al., n.d.):

1. Each variable is a column, each column is a variable
2. Each observation is a row, each row is an observation
3. Each value is a cell, each cell is a single value

Image source: Wickham et al. (n.d.) (all rights reserved)

• N.B., how you define a variable or observation is relative to what you want to do
  • for now, let’s consider a single trial per participant as an observation

the tidyverse

• a collection of R packages for tidy data
• you need to load a package at the beginning of every session
  • today we will mostly use functions from the dplyr package
    • if you load the tidyverse you don’t need to also load dplyr

# load tidyverse
library(tidyverse)

the magritrr pipe %>%

# take data frame and then...
iris %>%
  # print the head
  head()

  Sepal.Length Sepal.Width Petal.Length Petal.Width Species
1          5.1         3.5          1.4         0.2  setosa
2          4.9         3.0          1.4         0.2  setosa
3          4.7         3.2          1.3         0.2  setosa
4          4.6         3.1          1.5         0.2  setosa
5          5.0         3.6          1.4         0.2  setosa
6          5.4         3.9          1.7         0.4  setosa

load our data

# load lifetime data
readr::read_csv(here::here("data/data_lifetime_pilot.csv")) 

# A tibble: 4,431 × 28
   RECORDING_SESSION_LABEL TRIAL_INDEX EYE_USED IA_DWELL_TIME
   <chr>                         <dbl> <chr>            <dbl>
 1 px3                               1 RIGHT                0
 2 px3                               2 RIGHT                0
 3 px3                               3 RIGHT                0
 4 px3                               3 RIGHT                0
 5 px3                               3 RIGHT                0
 6 px3                               3 RIGHT                0
 7 px3                               3 RIGHT                0
 8 px3                               3 RIGHT                0
 9 px3                               4 RIGHT                0
10 px3                               5 RIGHT                0
# ℹ 4,421 more rows
# ℹ 24 more variables: IA_FIRST_FIXATION_DURATION <dbl>,
#   IA_FIRST_RUN_DWELL_TIME <dbl>, IA_FIXATION_COUNT <dbl>, IA_ID <dbl>,
#   IA_LABEL <chr>, IA_REGRESSION_IN <dbl>, IA_REGRESSION_IN_COUNT <dbl>,
#   IA_REGRESSION_OUT <dbl>, IA_REGRESSION_OUT_COUNT <dbl>,
#   IA_REGRESSION_PATH_DURATION <dbl>, KeyPress <dbl>, rt <dbl>, bio <chr>,
#   critical <chr>, gender <chr>, item_id <dbl>, list <dbl>, match <chr>, …

• was anything added to the Environment pane (top right box in RStudio)?

variable assignment with <-

• object_name <- code_output_to_be_saved_as_object_name

# load lifetime data and store it under df_lifetime
df_lifetime <- readr::read_csv(here::here("data/data_lifetime_pilot.csv"), 
                               # for special characters
                               locale = readr::locale(encoding = "latin1") 
                               ) 

• you should now see the object df_lifetime in the Environment pane

Tidyverse verbs

• verbs are functions from the tidyverse package
• for data tidying and transforming we’ll mostly use verbs from the dplyr package, which is part of the tidyverse
• check out RLadies Freiburg to see a YouTube video that covers most of these verbs

rename()

• one of the first things you’ll often want to do is rename some variables
• let’s start by re-naming some of our variables
  • e.g., RECORDING_SESSION_LABEL is a long way of saying ‘participant’

# rename variables
df_lifetime <- df_lifetime %>% # make df_lifetime from df_lifetime BUT THEN
  rename("px" = RECORDING_SESSION_LABEL, # rename a variable and (comma = 'and')
         "trial" = TRIAL_INDEX) # another variable

relocate

• the second step thing you might want to do is reorder your variables so the most important/relevant are near the beginning and ordered logically
  • let’s order our continuous reading time variables from ‘earliest’ to ‘latest’ measure

df_lifetime <- df_lifetime %>%
  relocate(ff,fp,rpd,tt, .after="eye") %>% 
  relocate(region_n, region_text, .after="trial")

names(df_lifetime[1:10])

 [1] "px"          "trial"       "region_n"    "region_text" "eye"        
 [6] "ff"          "fp"          "rpd"         "tt"          "fix_count"  

mutate()

Make some change

• new columns

df_lifetime <- df_lifetime %>%
  mutate(new_column = "new")

• change existing column

df_lifetime <- df_lifetime %>%
  mutate(new_column = px,
         trial = trial + 5)

• but let’s undo that…

df_lifetime <- df_lifetime %>%
  mutate(trial = trial - 5)

if_else()

• can be used inside mutate()
  • change values based on some logical condition
  • can be used to change an existing column, or create a new one
• ifelse(condition, output_if_true, output_if_false)

df_lifetime <- df_lifetime %>%
  mutate(new_column = if_else(name=="Aaliyah","name is Aaliyah","name is not Aaliyah"))

case_when()

• can be used inside mutate()
  • change values based on multiple logical conditions
  • can be used to change an existing column, or create a new one
• case_when(condition & other_condition | other_condition ~ output, TRUE ~ output_otherwise)
  • if you don’t include TRUE ~ output then NAs will created

df_lifetime <- df_lifetime %>%
  mutate(newer_column = case_when(
    name=="Aaliyah" & trial > 104 ~ "Aaliyah 2nd half",
    name=="Beyoncé" & (px == "px01" | px == "px04") ~ "Beyoncé px04 or px06",
    TRUE ~ "otherwise"))

Extra exercise

summary(as_factor(df_lifetime$region))

filler verb-1   verb verb+1 verb+2 verb+3 verb+4 
  1024    639    639    639    639    639    212 

names(df_lifetime)

 [1] "px"            "trial"         "region"        "region_n"     
 [5] "region_text"   "eye"           "ff"            "fp"           
 [9] "rpd"           "tt"            "fix_count"     "reg_in"       
[13] "reg_in_count"  "reg_out"       "reg_out_count" "rt"           
[17] "bio"           "critical"      "gender"        "item_id"      
[21] "list"          "match"         "condition"     "name"         
[25] "lifetime"      "tense"         "type"          "yes_press"    
[29] "KeyPress"      "new_column"    "newer_column"  "accept"       
[33] "accuracy"     

group_by() and ungroup()

Group data by certain variable(s)

• then perform some mutation
• then ungroup the data

df_lifetime <- df_lifetime |>
  group_by(px) |>
  mutate(px_accuracy = mean(accuracy)) %>%
  ungroup()

round(
  range(df_lifetime$px_accuracy),
  2)

[1] 0.26 0.90

select()

• keep only certain column(s)

df_lifetime %>%
  select(px)

# A tibble: 4,431 × 1
   px   
   <chr>
 1 px3  
 2 px3  
 3 px3  
 4 px3  
 5 px3  
 6 px3  
 7 px3  
 8 px3  
 9 px3  
10 px3  
# ℹ 4,421 more rows

df_lifetime %>%
  select(px, trial)

# A tibble: 4,431 × 2
   px    trial
   <chr> <dbl>
 1 px3       1
 2 px3       2
 3 px3       3
 4 px3       3
 5 px3       3
 6 px3       3
 7 px3       3
 8 px3       3
 9 px3       4
10 px3       5
# ℹ 4,421 more rows

Exercise

Remove the example variables we created with mutate:

• new_column and newer_column

# should look like this after
names(df_lifetime)

 [1] "px"            "trial"         "region"        "region_n"     
 [5] "region_text"   "eye"           "ff"            "fp"           
 [9] "rpd"           "tt"            "fix_count"     "reg_in"       
[13] "reg_in_count"  "reg_out"       "reg_out_count" "rt"           
[17] "bio"           "critical"      "gender"        "item_id"      
[21] "list"          "match"         "condition"     "name"         
[25] "lifetime"      "tense"         "type"          "yes_press"    
[29] "KeyPress"      "accept"        "accuracy"      "px_accuracy"  

filter()

• select certain rows based on certain criteria (==, !=, >, <, |)
  • N.B. when testing logical conditions == is needed

df_lifetime %>%
  filter(trial == 1)

# A tibble: 8 × 32
  px    trial region region_n region_text          eye      ff    fp   rpd    tt
  <chr> <dbl> <chr>     <dbl> <chr>                <chr> <dbl> <dbl> <dbl> <dbl>
1 px3       1 filler        1 He owned innumerabl… RIGHT     0     0     0     0
2 px5       1 filler        1 She is a mother of … RIGHT   145  1603  1603  1603
3 px6       1 filler        1 He is a father of t… RIGHT   147  1224  1224  1224
4 px2       1 filler        1 She made innumerabl… RIGHT    84  1829  1829  1829
5 px7       1 filler        1 In the '70s, he own… RIGHT   138  2456  2456  2456
6 px1       1 filler        1 Beloved morning sho… RIGHT   160  1708  1708  1708
7 px8       1 filler        1 She was a mother of… RIGHT   220   806   806   806
8 px4       1 filler        1 In the '70s, he own… LEFT    171  3557  3557  3557
# ℹ 22 more variables: fix_count <dbl>, reg_in <dbl>, reg_in_count <dbl>,
#   reg_out <dbl>, reg_out_count <dbl>, rt <dbl>, bio <chr>, critical <chr>,
#   gender <chr>, item_id <dbl>, list <dbl>, match <chr>, condition <chr>,
#   name <chr>, lifetime <chr>, tense <chr>, type <chr>, yes_press <dbl>,
#   KeyPress <dbl>, accept <dbl>, accuracy <dbl>, px_accuracy <dbl>

filter()

What are these code chunks doing?

df_lifetime %>%
  filter(px_accuracy > .5)

df_lifetime %>%
  filter(px == "px3")

df_lifetime %>%
  filter(px == "px3" | trial == "3")

df_lifetime %>%
  filter(px == "px3" & trial != "3")

Exercise

1. Create a new dataframe df_crit that includes only critical trials
2. Create a new dataframe df_fill that includes only filler trials

• Tip: trial type is stored in the column type

df_crit |> select(type) |> head()

# A tibble: 6 × 1
  type    
  <chr>   
1 critical
2 critical
3 critical
4 critical
5 critical
6 critical

df_fill |> select(type) |> head()

# A tibble: 6 × 1
  type  
  <chr> 
1 filler
2 filler
3 filler
4 filler
5 filler
6 filler

distinct()

df_crit %>%
  distinct(px)

# A tibble: 8 × 1
  px   
  <chr>
1 px3  
2 px5  
3 px6  
4 px2  
5 px7  
6 px1  
7 px8  
8 px4  

df_crit %>%
  distinct(px, name)

# A tibble: 639 × 2
   px    name          
   <chr> <chr>         
 1 px3   Edith Piaf    
 2 px3   Aaliyah       
 3 px3   David Beckham 
 4 px3   Jana Novotna  
 5 px3   Grace Kelly   
 6 px3   Nigella Lawson
 7 px3   Coco Chanel   
 8 px3   Ben Kingsley  
 9 px3   Jim Carrey    
10 px3   Judy Garland  
# ℹ 629 more rows

df_crit %>%
  distinct(px, name,
           .keep_all=T)

# A tibble: 639 × 32
   px    trial region region_n region_text eye      ff    fp   rpd    tt
   <chr> <dbl> <chr>     <dbl> <chr>       <chr> <dbl> <dbl> <dbl> <dbl>
 1 px3       3 verb-1        1 She         RIGHT     0     0     0     0
 2 px3       5 verb-1        1 She         RIGHT     0     0     0     0
 3 px3       8 verb-1        1 He          RIGHT     0     0     0     0
 4 px3      10 verb-1        1 She         RIGHT     0     0     0     0
 5 px3      13 verb-1        1 She         RIGHT     0     0     0     0
 6 px3      16 verb-1        1 She         RIGHT     0     0     0     0
 7 px3      18 verb-1        1 She         RIGHT     0     0     0     0
 8 px3      21 verb-1        1 He          RIGHT     0     0     0     0
 9 px3      23 verb-1        1 He          RIGHT     0     0     0     0
10 px3      26 verb-1        1 She         RIGHT     0     0     0     0
# ℹ 629 more rows
# ℹ 22 more variables: fix_count <dbl>, reg_in <dbl>, reg_in_count <dbl>,
#   reg_out <dbl>, reg_out_count <dbl>, rt <dbl>, bio <chr>, critical <chr>,
#   gender <chr>, item_id <dbl>, list <dbl>, match <chr>, condition <chr>,
#   name <chr>, lifetime <chr>, tense <chr>, type <chr>, yes_press <dbl>,
#   KeyPress <dbl>, accept <dbl>, accuracy <dbl>, px_accuracy <dbl>

arrange()

# default: ascending order (A-Z)
df_crit %>%
  distinct(px, trial, name, condition) %>%
  arrange(px, trial)

# A tibble: 639 × 4
   px    trial name            condition
   <chr> <dbl> <chr>           <chr>    
 1 px1       3 Amy Winehouse   deadPP   
 2 px1       5 John Wayne      deadPP   
 3 px1       8 Abraham Lincoln deadPP   
 4 px1      10 Helen Mirren    livingSF 
 5 px1      13 Paul McCartney  livingSF 
 6 px1      16 Ariana Grande   livingPP 
 7 px1      18 Kate Middleton  livingSF 
 8 px1      21 Johan Cruyff    deadSF   
 9 px1      23 Marilyn Monroe  deadPP   
10 px1      26 Biggie Smalls   deadSF   
# ℹ 629 more rows

# descending order (Z-A)
df_crit %>%
  distinct(px, trial, name, condition) %>%
  arrange(desc(px), trial)

# A tibble: 639 × 4
   px    trial name            condition
   <chr> <dbl> <chr>           <chr>    
 1 px8       3 Whitney Houston deadPP   
 2 px8       5 Elton John      livingSF 
 3 px8       8 Jackie Chan     livingPP 
 4 px8      10 Romy Schneider  deadPP   
 5 px8      13 James Cameron   livingSF 
 6 px8      16 Ella Fitzgerald deadSF   
 7 px8      18 Kathryn Hepburn deadPP   
 8 px8      21 Kate Middleton  livingPP 
 9 px8      23 Janis Joplin    deadPP   
10 px8      26 Serena Williams livingSF 
# ℹ 629 more rows

separate()

• create new columns from a single column

df_crit<- df_crit %>%
  separate(name,
           sep=" ",
           into = c("First","Last"),
           remove = F, # don't remove original column (name)
           extra = "merge") # if extra chunks, combine in 'Last' (von der...)

• opposite: unite()

Reshape data

• this is the major step of data tidying

  • make each column a variable
  • make each row an observation
  • make each cell a data point

• what variable and observation mean will depend on what you want to do, and will change at different steps of your analyses

• you typically want long data

  • but our dataset isn’t as long as it could be

pivot_longer()

• takes wide data and makes it longer
  • converts headers of columns into values of a new column
  • combines the values of those columns into a new condensed column
• takes a few arguments:
  • cols: which columns do we want to combine into a single column?
  • names_to: what should we call the new column containing the previous column names?
  • values_to: what should we call the new column containing the values from the previous columns?

pivot_wider()

• takes long data and makes it wider
• takes a few arguments:
  • id_cols: identifying columns
  • names_from: what should we call the new column containing the previous column names?
  • names_prefix:
  • values_from: new column values

Summary

• we saw that the equation for a straight line boils down to its intercept and slope

• we fit our first linear model with a categorical predictor

• next, we’ll look at a case with more than one predictor: multiple regression